There are various ways in which a human driver of a vehicle can interact with the external environment of the vehicle. For instance, the driver may communicate with other human drivers or pedestrians using various non-verbal gestures. A driver may waive a hand or make some other type of hand gesture. A driver may nod his or her head. Such non-verbal gestures can be used to express different feelings, make a request or signal an intended action. For instance, a non-verbal gesture can be used to indicate the direction that the driver intends to proceed or to express the driver's willingness to allow another driver or a pedestrian to take an action before the driver. Similarly, other human drivers or pedestrians in the external environment may communicate with the driver of the vehicle using one or more non-verbal gestures. Such non-verbal gestures can be used to communication various things, such as indicating that the vehicle driver should go ahead with his or her next maneuver or requesting that the vehicle driver take a certain action.